[Agentx][sglang] B200+B300 DeepSeek V4 AgentX config update#2145
[Agentx][sglang] B200+B300 DeepSeek V4 AgentX config update#2145Oasis-Git wants to merge 16 commits into
Conversation
Rework the B300 agentic sglang recipe into three regimes: - DP-attention (megamoe): MegaMoE DeepGEMM MoE (--moe-a2a-backend megamoe + mega_moe env + fused shared experts + autotune), mem-fraction 0.835, swa 0.075, prefill-delayer. Rank-adjusted sizing: DEP8 chunk 65536 / cuda-graph-max-bs-decode 544; DEP4 chunk 32768 / decode 128 (effective chunk 8192 for both). Measured DEP8 conc128: 24,466 -> 33,220 tok/s/gpu (vLLM v0.23.0 reference 28,962). - TP-only low-latency (TP8 or TP4, non-DP, conc <= 16): SGLang cookbook low-latency single-node recipe with speculative decoding removed: flashinfer_mxfp4 + --enable-deepseek-v4-fp4-indexer + fused shared experts, chunked-prefill 8192, mem-fraction 0.90. - TP8 mid concurrency (32-52): unchanged flashinfer_mxfp4 baseline. Builds on #2112 (image bump to nightly-20260707). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ecipe
B200 sglang agentic (dsv4_fp4_b200_sglang.sh):
- low-latency TP-only path (DP_ATTENTION=false): mem-fraction 0.90 (was 0.88),
matching the SGLang cookbook low-latency recipe (flashinfer_mxfp4, chunked
8192, swa 0.1, no spec, GPU-only).
- DEP path now splits by concurrency via DEP_HIGH_CONC:
conc < 54 -> conservative recipe (chunked 32768, mem 0.88, swa 0.1,
NUM_MAX_TOKENS_PER_RANK 4096, --cuda-graph-max-bs)
conc >= 54 -> cookbook high-throughput recipe (chunked 65536, mem 0.835,
swa 0.075, NUM_MAX_TOKENS_PER_RANK 8192,
--cuda-graph-max-bs-decode 544, --enable-prefill-delayer).
The 8192 tokens/rank cap keeps chunked 65536 on the DeepGEMM MoE path
instead of the fp4-incompatible Triton fallback.
configs/nvidia-master.yaml:
- dsv4-fp4-b200-sglang-agentic-hicache: refresh TP/DEP conc-lists.
- dsv4-fp4-b300-sglang-agentic-hicache: refresh conc-lists.
中文：B200 sglang 智能体脚本按并发拆分 DEP 配方（conc>=54 使用 cookbook 高吞吐配方，
含 chunked 65536 / prefill delayer / mem 0.835 / swa 0.075，并将 tokens/rank 上限提到
8192，避免 fp4 MoE 回退到 Triton 导致崩溃）；低延迟 TP-only 路径 mem 提到 0.90；
更新 master.yaml 中 B200/B300 sglang 智能体的并发列表。
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Oasis-Git
requested review from
Ankur-singh,
jgangani and
kedarpotdar-nv
as code owners
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
# Conflicts: # perf-changelog.yaml
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29081167653 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29081167653 |
cquil11
added
the
agentx
ChatGPT Codex Connector
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29081167653 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29081167653 |
| export SGLANG_OPT_USE_DEEPGEMM_MEGA_MOE=1 | ||
| export SGLANG_OPT_FIX_HASH_MEGA_MOE=1 | ||
| export SGLANG_OPT_USE_FAST_MASK_EP=1 | ||
| export SGLANG_OPT_FIX_MEGA_MOE_MEMORY=1 | ||
| export SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320 | ||
| export SGLANG_OPT_FIX_NEXTN_MEGA_MOE=1 | ||
| export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=0 |
There was a problem hiding this comment.
can u add these to sglang cookbook recipes
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29081167653 |
# Conflicts: # configs/nvidia-master.yaml # perf-changelog.yaml
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29308319290 |
# Conflicts: # perf-changelog.yaml
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29309998793 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29341637218 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29362826902 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29367944489 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29367944489 |
|
/reuse-sweep-run 29367944489 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29367944489 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29367944489 |
|
@claude Add a short, succinct, no fluff PR. description for posterity. |
|
Claude finished @cquil11's task in 46s —— View job Adding PR description
Added a PR description summarizing the DSv4 FP4 AgentX SGLang recipe changes: the B300 three-regime rework (MegaMoE DeepGEMM for DP-attention, flashinfer_mxfp4 low-latency for pure-TP conc ≤ 16, baseline otherwise), the B200 |
cquil11
changed the title
|
@Oasis-Git Here is the plan, it looks like there were some issues with B300, so I have opened a separate follow up pr #2226 In the meantime, we can merge this one since the B200 run completed successfully. In order to do this, we require the sglang cookbook is up to date with this PR. If any changes are needed to the B200 recipe, please open a PR against sgl cookbook and merge the PR and then update me (as well as linking PR here). |
|
Opened the corresponding SGLang cookbook update for the B200 DeepSeek-V4-Pro FP4 balanced recipe: sgl-project/sglang#31373 It aligns the major TP8/DP8 MoE backend with this PR by selecting MegaMoE and sets the 4096-token per-rank MegaMoE buffer required by the existing 32768-token DP8 prefill chunk. The cookbook retains its independently configurable EAGLE settings and leaves concurrency-sensitive memory and CUDA-graph sizing out of scope. |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29367944489 |
2 similar comments
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29367944489 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29367944489 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29367944489 |
4 participants
Tuning pass on the DeepSeek-V4 FP4 AgentX (agentic-coding) SGLang recipes for B200 and B300.
B300 (
dsv4_fp4_b300_sglang.sh)Reworked into three launch-time regimes:
--moe-a2a-backend megamoewith the DeepGEMM MegaMoE env stack,--enable-prefill-delayer,swa-full-tokens-ratio 0.075; rank-aware sizing — DEP8:mem-fraction-static 0.85,chunked-prefill-size 65536,cuda-graph-max-bs-decode 544; DEP4:0.93/32768/128.--moe-runner-backend flashinfer_mxfp4 --disable-flashinfer-autotune --enable-deepseek-v4-fp4-indexer,mem-fraction-static 0.90,page-size 256(SGLang DSV4 cookbook low-latency recipe; EAGLE spec decode omitted — unavailable here).flashinfer_mxfp4baseline.B200 (
dsv4_fp4_b200_sglang.sh)Split
mem-fraction-static: 0.88 under DP-attention (0.835 for the high-conc DEP arm), 0.90 otherwise.Config (
configs/nvidia-master.yaml)Refined B200/B300 agentic-hicache conc-list search spaces to resolve throughput cliffs and drop collapsed tail points.
perf-changelog.yamlupdated accordingly.